Skip to content

Update license scripts to support YAML frontmatter changes - #5259

Open
sarthak-shubham wants to merge 3 commits into
aboutcode-org:developfrom
sarthak-shubham:fix-yaml-frontmatter
Open

Update license scripts to support YAML frontmatter changes#5259
sarthak-shubham wants to merge 3 commits into
aboutcode-org:developfrom
sarthak-shubham:fix-yaml-frontmatter

Conversation

@sarthak-shubham

Copy link
Copy Markdown

Fixes #3138

What

Updated report_license_rules.py and buildrules.py in etc/scripts/licenses/
to work correctly with the YAML frontmatter format introduced in #3100, which
added new fields (is_builtin, is_generic, is_continuous, length) to the
License and Rule models.

report_license_rules.py

  • Added extrasaction='ignore' to the csv.DictWriter calls so the script no
    longer crashes when the models gain fields not present in the CSV column
    headers — also avoids needing to update this script every time a new model
    field is added going forward.
  • Added encoding="utf-8" to the open() call in write_data_to_csv(), since
    the default Windows encoding (cp1252) can't handle non-ASCII characters
    (e.g. in author names) present in the license database.
  • Added progress logging via click.echo() (start, periodic progress every
    5,000 rules, completion message) since exporting 35,000+ rules takes several
    minutes with no prior feedback that the script is working.
  • Switched to streaming with enumerate() instead of loading all rules into a
    list upfront, to support the progress logging without changing the output.

buildrules.py

  • Added a click.echo() message before the duplicate-check step
    (all_rule_by_tokens()). This step tokenizes all 35,000+ existing rules to
    check the new rule for duplicates, which takes roughly 10 minutes on a full
    run — expected given the dataset size, but silent without the log.

Testing

Ran both scripts manually against the current license/rule database —
report_license_rules.py completes end-to-end; buildrules.py correctly parses
an example rule file and generates a new .RULE file (deleted afterward to
avoid committing test artifacts).

I wasn't able to find existing automated tests covering these two scripts —
they appear to be maintainer-facing utility scripts outside the main src/
detection engine and its test suite. Is there a preferred location/pattern
for tests on scripts like these, or should I add something under tests/
following a different structure?

Note

Saw #3150 referenced in the issue as possibly covering part of this — happy
to adjust if there's overlap.

Also included

Added an AUTHORS.rst entry and a CHANGELOG.rst note per CONTRIBUTING.rst's
PR guidelines.

Signed-off-by: Sarthak Shubham <sarthakshubham23@gmail.com>
Signed-off-by: Sarthak Shubham <sarthakshubham23@gmail.com>
Signed-off-by: Sarthak Shubham <sarthakshubham23@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update license scripts for yaml frontmatter

1 participant